feat: Implement OpenTelemetry tracing #1741

iamKunalGupta · 2024-05-21T01:08:35Z

Add basic telemetry and examples
Integrate with temporal's opentelemetry interceptor

To run/test:

Run jaeger (it supports otlp format ingestion with a nice UI):

docker run --rm --name jaeger -p 16686:16686 -p 4317:4317 -e JAEGER_REPORTER_LOG_SPANS=true -e JAEGER_SAMPLER_TYPE=const -e JAEGER_SAMPLER_PARAM=1 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:1.57

Open jaeger ui at http://localhost:16686/

Run flow worker with the following env:

ENABLE_OTEL_TRACING=true
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317

Start any activity etc and we should start seeing traces like below:

TODOS:

add tracing to grpc+gateway

- Add basic telemetry and examples - Integrate with temporal's opentelemetry interceptor

flow/cmd/worker_common.go

flow/instrumentation/common/resource.go

serprex · 2024-05-21T01:26:45Z

flow/instrumentation/common/resource.go

@@ -0,0 +1,17 @@
+package common


avoid creating many small packages, it doesn't really matter until intersubpackage dependencies get involved. go's naming also makes having common names like util, utils, common, etc a pain

Already had to resolve util/utils split

the call graph is like setup -> tracing/metrics/logging -> common, so had to create a separate one this time.
I'll also change it to otel_common for better visibility

flow/instrumentation/setup.go

serprex · 2024-05-21T02:23:47Z

flow/connectors/core.go

+	ctx, span := trace.StartSpan(ctx, "connectors.GetCDCPullConnector")
+	defer span.End()


can move this into GetAs, using reflect.TypeFor[T] to put T in span label

CLAassistant · 2024-05-21T16:33:11Z

All committers have signed the CLA.

heavycrystal · 2024-05-27T15:51:46Z

flow/go.mod

@@ -112,12 +117,11 @@ require (
 	github.com/prometheus/common v0.53.0 // indirect
 	github.com/prometheus/procfs v0.15.0 // indirect
 	github.com/rogpeppe/go-internal v1.12.0 // indirect
-	github.com/segmentio/asm v1.2.0 // indirect
+	github.com/segmentio/asm v1.2.0 // indirect; indirectianian


is this tool generated?

feat: Implement OpenTelemetry tracing

76f948a

- Add basic telemetry and examples - Integrate with temporal's opentelemetry interceptor

iamKunalGupta requested a review from iskakaushik May 21, 2024 01:08